Purpose¶
- In this presentation, we'll briefly cover some of the methods that can be used to process and visualise data in Python
- This is intended to be a brief overview only - code examples are included, but will not be explained in great detail
Why Python?¶
- Python allows us to quickly and reproducibly process data
- The syntax of the language is (relatively!!) easy to understand
- Numerous modules exist which take out some of the hard work
Modules¶
- Python allows users to group together defined objects, variables and functions into a file (or several files) so that they can be used in other programs
- These are referred to as "modules"
- We will use several specialised modules which have to be installed separately by the user:
numpymatplotlib/seabornplotly
- We have to
importthese into our program
In [1]:
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set_style('darkgrid')
sns.set_context('poster')
import pandas as pd
import plotly
import plotly.graph_objs as go
plotly.offline.init_notebook_mode(connected=True)
Data Processing¶
Pandas¶
- Pandas is a Python module which describes itself as:
an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.
- It processes data quickly, and has good interactions with Excel and CSV files
- Generally, data is imported into a
DataFrame, which is a bit like an Excel worksheet - it uses columns and row to store data in a tabular format - Mostly we refer to columns by their names - note that using the correct case is critical (
sitenameis not the same asSiteName)
Reading in Data¶
- Reading in data from a CSV file is very straightforward
- We could read in all of the locations at once, but for now we'll pick just one
In [2]:
original_data = pd.read_csv('VBV_Data\VBV_20170131_JTC00567_shay_2940.csv', low_memory=False)
original_data.head(15)
Out[2]:
Processing Data¶
- Once we have the data imported into
pandas, we can start to manipulate it. - Python (and
pandas) can manipulate dates and times, if they're the correct kind of object. - We can convert the strings, and calculate hourly counts in a few lines of code
In [3]:
# Create a "Count" column, which will be 1 for each record to begin wth
original_data['Count'] = 1
# Make sure dates are used as dates, and set them as the index so we can summarise
original_data['DateTime'] = pd.to_datetime(original_data['RecordedDataTime'])
original_data.set_index('DateTime', inplace=True)
# Work out the vehicle count per class and direction for each hour
hourly_counts = original_data.groupby([pd.TimeGrouper('1H'), 'Info'])['Count'].count().reset_index()
hourly_counts.head(15)
Out[3]:
- We can add as many different columns as we need when grouping together our data:
In [4]:
# Work out the vehicle count per class and direction for each hour
hourly_counts_d = original_data.groupby([pd.TimeGrouper('1H'), 'Info', 'Direction'])['Count'].count().reset_index()
hourly_counts_d.head(15)
Out[4]:
Exporting Data¶
- Now that we have our data processed to provide the summaries we are interested in, we can export this summary back to a CSV as easily as we read the original data in
In [5]:
hourly_counts.to_csv('JTC00567 Hourly Summaries.csv', index=False)
hourly_counts_d.to_csv('JTC00567 Directional Hourly Summaries.csv', index=False)
- We can also export any number of
DataFrames to Excel
In [6]:
# Create a "writer"
excel_writer = pd.ExcelWriter('JTC00567.xlsx')
# Write each DataFrame to the writer separately, specifying the name we want to use for the worksheet
original_data.to_excel(excel_writer, 'Raw Counts', index=False)
hourly_counts.to_excel(excel_writer, 'Hourly Counts', index=False)
hourly_counts_d.to_excel(excel_writer, 'Hourly Counts_Directional', index=False)
# Save the "writer"
excel_writer.save()
Data Visualisation - matplotlib/seaborn¶
matplotlibis a plotting library for Python, allowing for the production of a wide range of graphsseabornextendsmatplotlibby adding support for further types of graphs, improved interaction withpandasdataframes and generally by makingmatplotlibgraphs prettier- By bringing
seaborninto our project, we use its colour schemes even when we plot graphs withmatplotlib(which in some cases is easier) - Any of the graphs shown here can be saved as an image file, allowing them to be used in reports or similar
Line Charts¶
matplotlibcan plot line charts from our data, and has a special interface for dealing with dates- Note that we don't have to specify the different classes - it loops over all of them automatically when we use
groupby
In [7]:
# Set up the graph size
fig, ax = plt.subplots(figsize=(15, 5))
# Get the data for each class in the DataFrame
for vehicle_class, data in hourly_counts.groupby('Info'):
# Plot the data for that class
ax.plot_date(data['DateTime'], data['Count'], label=vehicle_class, linestyle='-', marker='')
# Set up legend
plt.legend(frameon=True, facecolor='white',
loc='lower center', ncol=2, bbox_to_anchor=(0.5, -0.6))
# Set up axis labels
plt.xlabel('Date')
plt.ylabel('Vehicles')
# Fix margins properly
plt.ylim(bottom=0)
ax.margins(x=0)
plt.title('Total Flow - JTC00567')
Out[7]:
- Suppose we want to plot both directions separately - we can do this using
subplots
In [8]:
# Set up the subplots
fig, all_axes = plt.subplots(2, figsize=(15, 10), sharex=True, sharey=True)
# Get the data for each class in the DataFrame
for i, (direction, directional_data) in enumerate(hourly_counts_d.groupby('Direction')):
for vehicle_class, data in directional_data.groupby('Info'):
# Plot the data for that class
all_axes[i].plot_date(data['DateTime'], data['Count'], label=vehicle_class, linestyle='-', marker='')
all_axes[i].margins(x=0)
all_axes[i].set_title(direction)
all_axes[i].set_ylabel('Vehicles')
# Set up legend
plt.legend(frameon=True, facecolor='white',
loc='lower center', ncol=2, bbox_to_anchor=(0.5, -0.7))
# Set up labels
plt.xlabel('Date')
fig.suptitle('Flow by Direction - JTC00567')
# Fix margins properly
plt.ylim(bottom=0)
Out[8]:
Histograms¶
- Suppose we want to plot a histogram of the car speeds
pandasincludes a method that lets aDataFramebe exported as a histogram
In [9]:
# Set up the graph size
fig, ax = plt.subplots(figsize=(15, 5))
# Filter our original DataFrame to only contain cars (class 2)
cars = original_data[original_data['Class'] == '2']
# Set up the bins we want to use in our histogram
bin_size = 10
bins = range(0,max(cars['SpeedMPH']) + bin_size, bin_size)
# Plot the histogram
cars.hist(column='SpeedMPH', bins=bins, ax=ax, alpha=0.5)
# Set up labels and title
plt.xlabel('Speed (mph)')
plt.ylabel('Number of Vehicles')
plt.title('Car Speeds Histogram')
# Fix the margins
ax.margins(x=0)
Heatmaps¶
- One of the advantages of
seabornovermatplotlibis the ease with which heatmaps can be made - We can simply provide the rows, columns and value we want to represent
- Suppose we want to create a heatmap of our car flows - each row representing a day and each column an hour
- We need to adjust our data into the correct format - first we'll filter to just cars and add in some nice columns for the date and time
In [10]:
nb_cars = hourly_counts_d[(hourly_counts_d['Info'] == 'Car / Small Van <5.2 metres') &
(hourly_counts_d['Direction'] == 'N')].copy()
nb_cars['Day'] = nb_cars['DateTime'].dt.weekday_name
nb_cars['Hour'] = nb_cars['DateTime'].dt.time#hour.astype('str')
nb_cars.head(15)
Out[10]:
- Next, we'll pivot this toget the days as rows and hours as columns.
In [11]:
column_order = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday']
pivot = nb_cars.pivot(index='Day', columns='Hour', values='Count')
pivot = pivot.reindex_axis(column_order, axis=0)
pivot
Out[11]:
- From this data, creating a heatmap is very simple
In [12]:
# Set up the graph size
fig, ax = plt.subplots(figsize=(15, 5))
sns.heatmap(pivot)
Out[12]:
- If we choose, we can annotate this heatmap with the counts
In [13]:
# Set up the graph size
fig, ax = plt.subplots(figsize=(15, 5))
sns.heatmap(pivot, annot=True, fmt='d', annot_kws={'size': 12})
Out[13]:
Box and Whisker Plots¶
- Box and whisker plots are good for showing the distribution of data
- For this example, let's collect the data for every site observed
- As different sites use different classifications, we'll filter to car data only by using a partial match on the word "Car"
In [14]:
import os
all_data = pd.concat([pd.read_csv(os.path.join('VBV_Data', f), low_memory=False)
for f in os.listdir('VBV_Data')
if f.startswith('VBV')])
all_cars = all_data[all_data['Info'].str.contains('Car')]
- Next, we again use seaborn directly to create a box plot
In [15]:
# Set up the graph size
fig, ax = plt.subplots(figsize=(15, 8))
box_plot = sns.boxplot(data=all_cars, y='SpeedMPH', x='SiteName')
for lbl in box_plot.get_xticklabels():
lbl.set_rotation(90)
ax.margins(y=0)
Data Visualisation - plotly¶
plotlyis a different data visualisation library that allows for the creation of interactive graphs- This can be useful for clients who enjoy being able to interrogate the underlying data themselves
- However, they need to be provided as HTML files, which may unfamiliar for some clients, and can be slow to load if a lot of data is involved in the plot.
Line Charts¶
- Standard line charts are similar to the process followed in
matplotlib
In [22]:
lines = []
for vehicle_class, data in hourly_counts.groupby('Info'):
# Set up the line plot for each vehicle type, and append to a list
line = go.Scatter(x=data['DateTime'], y=data['Count'], name=vehicle_class)
lines.append(line)
# Set up layout
layout = go.Layout(legend=dict(orientation='h',
xanchor='center',
y=-0.2,
x=0.5),
xaxis=dict(title='Date'),
yaxis=dict(title='Vehicle Count'),
autosize=False,
width=1000,
height=600)
# Combine lines and layout, and plot
fig = go.Figure(data=lines, layout=layout)
plotly.offline.iplot(fig)
- Showing the two directions separately side-by-side with this much interactivity might be a bit of an information overload
- We can reduce this slightly by using a drop-down menu to let the user choose the direction they want to view.
In [31]:
# Set up empty lists
lines = []
type_counts = []
for direction, directional_data in hourly_counts_d.groupby('Direction'):
# Initialise counter
i = 0
for vehicle_class, data in directional_data.groupby('Info'):
# Set up the line plot for each vehicle type, and append to a list
line = go.Scatter(x=data['DateTime'], y=data['Count'], name=vehicle_class,
visible=not len(type_counts))
lines.append(line)
# Increment counter
i += 1
# Add the number of types for this direction to the list
type_counts.append(i)
# Set up update menus
updatemenus = list([
dict(active=0,
showactive=True,
buttons=list([
dict(label = 'N',
method = 'update',
args = [{'visible': [True] * type_counts[0] + [False] * type_counts[1]},
{'title': 'N'}]),
dict(label = 'S',
method = 'update',
args = [{'visible': [False] * type_counts[0] + [True] * type_counts[1]},
{'title': 'S'}])
])
)
])
# Set up layout
layout = go.Layout(legend=dict(orientation='h',
xanchor='center',
y=-0.2,
x=0.5),
xaxis=dict(title='Date'),
yaxis=dict(title='Vehicle Count'),
autosize=False,
width=1000,
height=600,
updatemenus=updatemenus)
# Combine lines and layout, and plot
fig = go.Figure(data=lines, layout=layout)
plotly.offline.iplot(fig)
Histograms¶
- Again, the process is fairly similar to
matplotlib
In [23]:
# Set up the histogram
hist = go.Histogram(x=cars['SpeedMPH'], opacity=0.5,
autobinx=False, xbins=dict(start=0,
end=max(cars['SpeedMPH']),
size=10))
# Set up the layout
layout = go.Layout(xaxis=dict(title='Speed (mph)'),
yaxis=dict(title='Number of Vehicles'),
title='Car Speeds Histogram',
autosize=False,
width=1000,
height=600)
# Combine the histogram and layout
figure = go.Figure(data=[hist], layout=layout)
plotly.offline.iplot(figure)
Heatmaps¶
- The heatmap is slightly different - the matrix format we used for
matplotlibwon't work here.
In [18]:
# We need to sort our data so the y axis is shown in the correct order
column_order = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday']
nb_cars['Day'] = pd.Categorical(nb_cars['Day'],
categories=column_order,
ordered=True)
nb_cars.sort_values(['Day', 'Hour'], inplace=True)
# Generate the heatmap
heat = go.Heatmap(x=nb_cars['Hour'],
y=nb_cars['Day'],
z=nb_cars['Count'],
hoverinfo='z')
# Generate the layout
layout = go.Layout(xaxis=dict(title='Hour'),
yaxis=dict(autorange='reversed'),
title='Car Counts Heatmap',
autosize=False,
width=1000,
height=600)
# Combine the data and layout
figure = go.Figure(data=[heat], layout=layout)
plotly.offline.iplot(figure)
Box and Whisker Plots¶
- We have to set up the box plots individually but otherwise the process is similar
- This graph is on the next slide as it may take a while to load
In [21]:
# Set up the individual boxes
boxes = [go.Box(y=data['SpeedMPH'], name=site, boxmean=True)
for site, data in all_cars.groupby('SiteName')]
# Set up the layout
layout = go.Layout(xaxis=dict(title='Site'),
yaxis=dict(title='Speed (mph)'),
title='Car Speeds Box Plot',
autosize=False,
width=1000,
height=600)
# Combine the data and layout
figure = go.Figure(data=boxes, layout=layout)
plotly.offline.iplot(figure)
Summary¶
- This presentation has covered only the basics of data processing and just a few examples of the graphs that can be produced with these libraries
Pandashas many further functions, and there are a lot of other libraries to use in data processing, including database libraries such assqlite3- There are also various other libraries available that do similar things, such as
BokehandHoloViews